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INTRODUCTION 

As the responsible agency for administering the U.S. Department of 
Agriculture's Food Stamp Program, a major priority for the Food and Nutrition 
Service (FNS) is to reduce losses to the program due to fraud and error. It 
has become increasingly difficult to develop strategies that are effective in 
reducing errors. As the error rate decreases nationally, there remains a very 
costly set of errors that represent a relatively small percentage of total 
cases. Further, these errors are distributed heterogeneously, and they are 
elusive to methods that have been effective in the past. FNS has sponsored 
several initiatives to reduce errors. One of the most significant resources 
for error reduction strategies is state and local food stamp agencies. By 
virtue of having responsibility for service delivery, state and local agencies 
have firsthand knowledge of the systems that allow errors to occur. This 
knowledge cannot be duplicated within FNS or by external sources. 

In July of 1983, USDA solicited state and local agencies to submit 
proposals for demonstrations and evaluations designed to reduce errors and 
abuse in the Food Stamp Program through fraud prevention and/or detection 
strategies and improved management practices. Authorization for this 
solicitation is contained in Section 17 of the Food Stamp Act, as amended. 
FNS established cooperative agreements with three states — Maryland, North 
Carolina, and Vermont— as a result of this solicitation. The Food and 
Nutrition Service has funded other error reduction demonstrations previously, 
and other Federal agencies fund similar demonstrations (e.g., the Department 
of Health and Human Services funds states to demonstrate error reduction 
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strategies in the Aid to Families with Dependent Children Program). The 
purpose of this paper is to review the state demonstrations from a management 
perspective and in particular to discuss: 

• Opportunities in state initiated demonstrations 

• Pitfalls in state initiated demonstrations 

• The tension between Federal and State priorities 

• The role of FNS' technical assistance contractor 

• Recommendations for designing, implementing, and evaluating error 
reduction projects involving state agencies and the Federal 
government. 

1 . OPPORTUNITIES FOR STATE ERROR REDUCTION DEMONSTRATIONS 

As the Federal government struggles with the substantial deficit, 
increased emphasis is placed on assuring quality in the delivery of Federal 
benefits to individuals. In the past fifteen years the Food Stamp and Aid to 
Families with Dependent Children programs have cut the rates of benefit 
mi sal locations in half. The remaining error is more heterogeneous and more 
elusive. For this reason the Federal government has taken advantage of the 
opportunity to draw upon the perspectives and experiences of states in 
identifying error reduction strategies. States vary in the nature of their 
error. They are consistent, however, in a broad developmental strategy - 
first attacking the greatest error source(s) in the most direct and 
administratively feasible fashion, followed by more targeted efforts on the 
greatest remaining sources of error. States also initially tend to 
internalize responsibility and accountability at the state level, and later 
communicate the sense of responsibility to the local offices and workers and 
to the beneficiaries themselves. At this point, most states are taking 
actions that foster accountability at the local office, individual caseworker, 
and client levels. Because each state has a unique history of prior actions 
and operates in a different political and economic framework, the state 
perspective in error reduction is very valuable to the Federal government. As 
a state builds upon its history to attack the remaining problems, it has very 
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valuable lessons to share with other states that have similar problems and a 
potential for similar solutions. The ideas for error reduction generated at 
the state level, therefore, are an important asset in reducing Federal program 
costs . 

In carrying out demonstrations and evaluations, the states face the daily 
challenges of testing a strategy and a hypothesis in a real world environ- 
ment. The state's experiences in charting a course through a variety of 
contextual impediments (such as the strong urge to adopt a strategy with face 
validity on a statewide basis; modifying local office indifference; allocating 
already tight resources to the project) also provide valuable lessons for 
other states that are considering similar actions. 

Carrying out such demonstrations helps to build states' capacity for 
basing management decisions on rigorous empirical evidence. State agencies by 
definition are primarily deliverers of service. State agencies, as well as 
other levels of government, base most policy decisions and management actions 
on some combination of empirical evidence and political viability. In an 
environment of tight resources, it is more prudent to place greater value on 
empirical evidence, particularly evidence indicative of cost effectiveness; 
before embarking on an action that may be costly, but ineffective. Conducting 
demonstrations gives states valuable exposure to the rigors of research 
methodology and establishes empirical evidence as a basis for decision-making. 

Taken together, state initiated efforts to demonstrate error reduction 
strategies are very beneficial to the states and to the Federal government. 
Such projects serve as models for empirically-based management decision-making 
and emphasize cost effectiveness at the state level. They develop skills that 
better equip states to take charge of the problem of errors. And, in turn, 
they save the Federal government money by enhancing the states' capacities to 
reduce errors. 
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2. PITFALLS IN STATE-INITIATED ERROR REDUCTION DEMONSTRATIONS 



While the concept of state initiated demonstrations is very appealing, 
there are two major problems associated with state implementation of Federally 
funded demonstrations: 

• State capabilities to conduct demonstrations and evaluations 

• Conflicts between Federal and State goals and roles. 

The following sections expand on issues in these areas. 

2.1 State Capabilities to Conduct Demonstrations 

The conduct of demonstrations and evaluations by state agencies requires 
service delivery experts to function as research experts. In reviewing 
Federally funded evaluations, it has been observed that state service delivery 
agencies typically do not have strengths in designing and conducting rigorous 
demonstrations and evaluation. The research role in service delivery is 
typically to produce statistics for monitoring performance and to describe 
trends in the population of beneficiaries. Very little state research is 
initiated with state funds. Since acting on a small (from a statistical 
perspective) error rate creates many complex design problems that require 
sophisticated research expertise, many state agencies do not possess the 
caliber of expertise for dealing with complex problems associated with 
selecting a sound treatment, designing a demonstration that can produce 
observable results, and designing an appropriate evaluation methodology. The 
review of Federally funded demonstrations revealed: 

• Difficulties in Assessing Potential Treatment Impact . A state 
proposed a treatment that would reduce error by 50 percent. This was 
impossible because the aspect of error that the treatment targeted, 
i.e., income errors or the time of application, only accounted for 30 
percent of the states' error. 

• Problems in Developing Effective Treatments . Most states did not 
conduct a vigorous analysis of their error data to develop a treat- 
ment. Rather, treatments were proposed because they had face 
validity as one of many potential solutionsto the general problem or 
error. 
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Inadequate Statistical Designs . Most states proposed sample sizes 
that yielded inadequate statistical power to detect an impact. 
Consequently, real impacts would go undetected in the states' designs. 



• Implementation Problems . Many states failed to recognize the time 
and effort needed to secure cooperation or data from other state 
agencies or from local offices. Project budgets and schedules were 
increased because of such problems. States encountered delays in 
assigning or acquiring staff to conduct demonstrations. The types of 
problems included hiring freezes, state policies that required staff 
to take leave from a "tenured" position to participate in the 
demonstration, and risk not getting the old job back, and the 
bureaucratic delays associated with hiring contract staff or 
subcontractors . 

Taken together, state service delivery agencies and the bureaucracies in 
which they exist, are not the most appropriate and efficient site for 
conducting demonstration and evaluation research. 

2.2 Tension Between Federal and State Roles and Priorities 

Previous Section 17 solicitations released by the Food and Nutrition 
Service (FNS) resulted in grants to state and local agencies. The 1983 
solicitation, however, specified cooperative agreements as the type of 
arrangement for funding these projects. A key factor in the decision to use 
the cooperative agreement as the legal arrangement was based on FNS' 
perception that the lack of documented success in previous grant projects 
could be improved by using a contractual medium that allows FNS to exercise 
more control over the direction of the project than in a grant. Under a grant 
arrangement FNS provided funding for projects but allowed grantees maximum 
control over the technical direction of the project and the development of 
interim products. The major distinction between the cooperative agreement and 
previous grants was operationalized as "go/no-go decision" points at key 
milestones during the life of the project. These decision points allow FNS to 
review the contractor's work and request revisions, as necessary, to assure 
that the project meets FNS' quality standards and shows promise of a 
successful conclusion. 
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Also included in the cooperative agreement was the requirement for the 
cooperators to accept assistance from FNS' evaluation assistance contractor. 
The evaluation contract was made available by FNS to provide assistance in 
areas where difficulties might arise, especially in two areas observed to be 
weaknesses in previous grants — evaluation design and analysis. 

The use of the cooperative agreement coupled with the availability of an 
evaluation assistance contractor was expected to alleviate weaknesses 
experienced in the previous grant program. However, even with these remedies 
in place, the three fraud and error reduction projects funded by FNS in 1983 
have not proceeded smoothly. In every case, it was necessary for FNS to 
extend the design and development phase to assure reasonable demonstration and 
evaluating designs, and in every case the cooperators have requested increased 
funding. Many of the problems that have occurred are systemic in a situation 
where a state agency is carrying out a demonstration and evaluation project. 

There are conflicts between the goals FNS has for the projects, the goals 
of the state, and the goals of the state's subcontractor. FNS' goals have 
been most clearly articulated, both stated in the official announcement 
soliciting proposals and formally and informally on numerous occasions. FNS' 
overall goals are to ensure that the demonstrations are successful, evaluable, 
and transferable. The state goals stated in the proposal were generally that 
the project show evidence of error reduction and/or increased efficiency. 
Subcontractor goals were never articulated. In one state, it appears that the 
subcontractor goals are first to support the state goals and second to support 
FNS' goals. In another state, the subcontractor's goals seem related to 
capitalizing on the entrepreneurial aspects of the project, e.g., developing 
an in-home audit/quality control review capability and a computerized 
interview product that the subcontractor can market to other counties within 
other states. 

Many of the problems that have occurred in the demonstrations could be 
attributed generally to the area of goal conflict and to the associated roles 
that states see for themselves and the standards by which the various agencies 
involved assess achievement of the goals. 
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In general, while all states proposed projects to meet FNS' goals, they 
operational i zed the projects to meet state goals. In one state, there were 
attempts to reduce statewide error by applying the treatments statewide and 
rendering FNS' evaluation goals impotent. In a second state, continued 
emphasis was placed on collecting outcome data that were inadequate for 
hypothesis testing, but would carry strong political weight within the state. 
Athird state maintained a strong desire to use the project to fund fraud 
workers even though there was little logical connection between the treatment 
and an increase in fraud. Essentially, the project funded the state to reduce 
its backlog of fraud referrals. 

The issues of role conflict is inevitable given tight resources within all 
the states and the overriding concern for the states to reduce errors. The 
impact on the demonstrations is that Federal goals will inevitably be diluted 
when the states conduct the demonstrations. The degree to which FNS is 
willing to accept this dilution should influence FNS' decisions to continue to 
award these types of projects to states regardless of the contractual vehicle. 

3. ROLE OF FNS' ASSISTANCE CONTRACTOR 

In the current projects, the role of the assistance contractor was not 
defined specifically at the beginning of the projects. Although it was 
recognized that the states themselves, being service delivery agencies, were 
not well equipped to handle all aspects of the projects, it was not possible 
to anticipate specific weaknesses, if any, in the states' contract staffs' and 
subcontractors' capabilities to carry out the projects. It was mutually 
accepted by FNS and Applied Management Sciences that states would be alienated 
if FNS and Applied Management Sciences "imposed" assistance. Therefore, 
states were asked to articulate their needs. In general, states did not 
perceive that they had assistance needs. One of the reasons that this 
occurred was that states did not recognize their (including contract staff and 
subcontractors) shortcomings relative to FNS' goals and standards for the 
project. States tended to focus on the project fulfilling their own needs and 
managing the project accordingly. 
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The role of the assistance contractor evolved to become an extension of 
FNS in its monitoring function by reviewing and critiquing the work of the 
state project staff. Applied Management Sciences also helped states to 
articulate their designs and make revisions to the products produced by states 
when the projects failed to meet FNS standards. Since states have responsibi- 
lity to carry out the project. Applied Management Sciences was not generally 
used as a substitute to compensate for an anticipated deficiency, but was 
called upon after a deficiency became obvious. One of the impacts of this 
approach was extensive delays in state preparation of their design plans. 
After receiving Applied Management Sciences' critiques, states attempted to 
prepare responsive design papers. The combination of a lack of technical 
knowledge and prominence of the state goals resulted in inadequate plans being 
revised numerous times. A more efficient approach would have been for Applied 
Management Sciences to work as co-authors with the demonstration staff in 
preparing the paper, instead of functioning in a role of providing advice and 
critiques that are then interpreted by the demonstration staff. This example 
can be extended to a general role of co-participant for the assistance 
contractor instead of a monitoring and advisory role. To be effective, this 
role must be stated up-front, and not imposed later. 

There is, of course, the pitfall that the state may abdicate responsibi- 
lity for the project and come to rely entirely on FNS' assistance contractor. 
Another liability might be the state's resistance to FNS' contractor assuming 
some of the ownership for the project. However, regardless of these pitfalls, 
FNS would be assured of acceptable products produced on a timely basis. 

4. CONCLUSIONS AND RECOMMENDATIONS 

Given these experiences and conditions, how should FNS design the 
demonstrations as authorized by Section 17 of the Food Stamp Act so that they 
capitalize on states' assets and circumvent weaknesses? To date, FNS has 
funded states as grantees, as cooperators, and has provided assistance via a 
contract research firm. FNS has not funded states as contractors, and this is 
an option to FNS to consider. This would maximize FNS' control over the 
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effort by providing for more products and milestones to be reviewed and 
accepted. Theoretically, this is an ideal solution. Realistically, the 
systemic problems discussed previously will still manifest themselves in a 
contract, and it is likely that friction between FNS and states will be 
exacerbated, and that FNS, in its role of contracting authority, will be 
forced to issue stop work orders and cancellations. Further, states generally 
do not have technical staff qualified to assume responsibility for a contract 
and the states' systems were not designed to support stringent contractual 
requirements for monitoring costs and milestones or producing acceptable 
products. The use of a contract vehicle would likely be a barrier to 
participation for many states that realistically assess their capacity to 
fulfill contractual requirements. 

The basic conclusion that states are not equipped technically to conduct 
demonstrations and evaluations that meet FNS standards leads to two general 
conclusions: 

• FNS should be realistic in its expectations for state performance 

• FNS should build supports into the process that help to assure 
effectiveness and that FNS' goals are met. 

It is not realistic to expect that a state can carry out a demonstration 
and evaluation as effectively and efficiently as organizations whose major 
service is research and evaluation. Consequently, it is a given that state 
demonstrations will take longer than and not run as efficiently as contracts 
that FNS funds. To take advantage of the states' unique knowledge, and to 
support states in their capacity for basing decisions on an empirical base, 
FNS should restructure the management and administration of state level 
projects. 

The cooperative agreement is the most appropriate contractual vehicle. 
Grants offer FNS little control and many states would not be able to fulfill 
the obligations of a contract. The cooperative agreement, however, should be 
enhanced in three ways: 

• First, FNS' goals should be more clearly articulated in the agreement 
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Second, a kick-off workshop should be held to review the generic 
lessons learned and problems from previous projects 



• Third, the sequence of the projects should have a more intense design 
phase, followed by full implementation, if appropriate. 

The cooperative agreement should be expanded to describe specifically FNS' 
extent of involvement, and the requirements of the cooperator. At a minimum, 
this should include: 

• Required tasks that the cooperator must perform: 
Phase I: Develop Treatment 

Task 1.1: Attend Workshop on Design . 

Task 1.2: Review Historical Rationale . Prepare an historical 
statement of rationale that specifies how the proposed 
project builds upon the state's previous efforts to reduce 
errors, and why it is the logical next step. 

Task 1.3: Define Contextual Rationale . Prepare a statement of 
rationale that discusses how the proposed project builds 
upon efforts initiated in other states. 

Task 1.4: Define Empirical Rationale . Conduct an empirical 
analysis, preferably of QC data, to identify error sources 
and provide an empirical rationale for a treatment. 

Task 1.5: Develop Treatment Design . Based upon outputs of Tasks 1, 
2, and 3, propose an effective (set of) treatment(s) . 

Task 1.6: Confer with FNS Support Contractor in Developing the 
Rationale and Treatments . Specify consultation schedule 
and activities of support relative to all Phase I tasks. 

Task 1.7: Brief FNS on Treatment Design . 

Phase 2: Develop Demonstration Design 

Task 2.1: Attend Workshop on Demonstration Design . 

Task 2.2: Articulate the Demonstration . Specify the operational 
aspects of the demonstration. 

Task 2.3: Identify Kev Participants . Contact all involved parties 
and agencies and provide them with statements of 
participation requirements defining: action, timing, 
potential impacts on ongoing operations. Obtain signed 
agreements to participate and support the project. 
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Task 2.4: Articulate Demonstration Management Plan and Schedule . 

Task 2.5: Confer with FNS Support Contractor . Specify consultation 
schedule and activities of support relative to all Phase 
II tasks. 

Phase 3: Develop Evaluation Design 

Task 3.1: Attend Workshop on Evaluation Design . 

Task 3.2: Develop the Sample . Establish sample frame and sample 
sizes necessary to detect impacts. 

Task 3.3: Develop Impact Measures . Design instruments and/or other 
data sources. 

Task 3.4: Develop Analysis Plan . Specify hypotheses, data inputs, 
proposed analyses and expected outcomes. 

Task 3.5: Prepare Evaluation Methods Design . Use outputs of 
preceding tasks to prepare evaluation design. 

Task 3.6: Develop Evaluation Management Plan and Schedule . 

Task 3.7: Confer with FNS Support Contractor . (All Phase 3 tasks.) 

Phase 4: Implement Demonstration 

Task 4.1: Attend Workshop on Implementation Pitfalls . 

Task 4.2: Enhance Phase I Operational Plans on the Basis of Pilot 
Test. New Information . 

Task 4.3: Carry Out Demonstration and Report at Least Biweekly on 
Activities . 

Task 4.4: Collect Data for Evaluation . 

Task 4.5: Confer with FNS Support Contractor . (All Phase 4 tasks.) 
Phase 5: Assess Impact 



Task 5.1 
Task 5.2 
Task 5.3 
Task 5.4 
Task 5.5 



Attend Workshop in Impact Assessment . 
Prepare Data for Analysis . 
Analysis Data . 
Report Results . 

Confer with FNS Support Contractor . (All Phase 5 tasks.) 
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• Contributions from FNS and FNS' contractor: 

a. Access to QC data and support in analyzing data 

b. 1-3 day workshops on all important project phases 

c. Consultation from FNS' support contractor in all aspects of the 
project to assure that FNS' goals are met 

d. Teleconferences to facilitate cooperation and participate in 
problem resolution 

e. Site visits to facilitate cooperation, monitor performance, and 
participate in problem resolution 

f. Financial support to carry out the project. 

• Cooperator's contributions: 

a. Administrative Director, representing state agency's commitment 
to the demonstration, and facilitation of access to other state 
participants. 

b. Technical director representing both necessary management and 
research and evaluation skills necessary to carry out the 
project. Contract staff or subcontractor may serve as deputy 
director, if desired, to provide technical expertise. 

c. Fulfill task requirements by providing necessary staff, 
facilities, etc. 

• Areas of Cooperation: 

FNS and the cooperator must work together to meet mutually agreeable 
project goals. Consultations, as appropriate, will occur at decision 
points throughout each phase of the project. 

The cooperative agreement's up-front specificity of roles and requirements 
will foster greater cooperation. The contractor-developed workshops will 
alert states to likely pitfalls and better enable them to recognize shortfalls 
and assistance needs. 

In this model, FNS would solicit states to propose an idea for an 
effective error reduction strategy. This could be achieved in a 10 to 20 page 
concept paper. Depending on response, all or a subset of the ideas would be 
selected for a design and development project. Selection would be based on 
judgment as to innovativeness, strength of rationale, potential for success. 
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evaluability, transferability. FNS would fund states to cooperate with a 
contractor under FNS' control to develop appropriate treatments. States would 
cooperate by making data available and by being available to meet and brain- 
storm with the contractor. The contractor would be responsible for working 
with the states to articulate a treatment, develop a demonstration design, and 
propose an appropriate evaluation design (Phases I through III described 
earlier). The experience of working with a contractor to articulate solutions 
and develop appropriate corrective actions would be very beneficial to the 
states. It would build state staff capacity in how to go about identifying 
and selecting appropriate corrective actions, which is a role all have been 
forced to assume, but few are prepared to assume effectively. 

The plan, once approved by the state, would be submitted competitively to 
FNS for funding. This funding would result in a second cooperative agreement 
with the state for working with FNS' contractor to carry out the demonstration 
and evaluation (Phases IV and V). The specific involvement of the contractor 
would be determined on an individual basis, depending on the state's capacity 
to conduct various aspects of the project. Hence, an important effort for FNS 
will be to make a comprehensive assessment of the state's capacity to carry 
out the project. This will entail: 

• State level of commitment and leadership 

• Sophiostication and integration of the support systems (e.g., MIS) 
needed for the project 

• Extent of primary data collection needed and resources necessary to 
obtain it successfully 

• Proven cooperative relationships with other affected offices and 
agencies 

• Analytic capabilities. 

At a minimum, full time state staff in areas affected by the project would 
serve on task forces or oversight committees. At a maximum, state staff would 
participate in conducting the demonstration and evaluating effectiveness. 
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In this model, FNS would be able to take advantage of the unique knowledge 
and skills that the state brings to the problem. To assure achievement of its 
goals, FNS would have direct control over the contractor's performance. The 
contractor would be accountable to FNS, but would also report to a state 
oversight committee to assure state involvement in key issues. 
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